<html>
<head><meta charset="utf-8"><title>davidtwco&#x27;s memory issue · general · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/index.html">general</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html">davidtwco&#x27;s memory issue</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="128592995"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128592995" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128592995">(Jun 25 2018 at 10:17)</a>:</h4>
<p>Not sure if this is the best place to ask, it's unrelated to any Rust contributions I'm working on - I've got a project where the memory usage keeps growing when run with multiple threads. The same code without threads uses a constant amount of memory, running that code on multiple threads - even though they don't interact or share any data - causes the memory to keep increasing. </p>
<p>I've made a small-ish example (<a href="https://gist.github.com/davidtwco/46b9a212654c9b8d8061df056467a20d" target="_blank" title="https://gist.github.com/davidtwco/46b9a212654c9b8d8061df056467a20d">https://gist.github.com/davidtwco/46b9a212654c9b8d8061df056467a20d</a>) but it requires you have multiple Jenkins instances running with a bunch of jobs in each - so it's not really all that useful. I've checked it on three different machines with two different OS' and I've included a massif output file since it might be hard to run it. It seems to be all <code>serde_json</code> making the allocations, but I can't see why it wouldn't be getting dropped - it's entirely possible (and probably likely) that I've just made a mistake somewhere.</p>
<p>Would appreciate any thoughts anyone has - been staring at this for a while with no luck.</p>



<a name="128599163"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128599163" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128599163">(Jun 25 2018 at 13:12)</a>:</h4>
<p>Are more and more threads spinning up?</p>



<a name="128599241"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128599241" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128599241">(Jun 25 2018 at 13:14)</a>:</h4>
<p>My first thought is memory fragmentation; memory usage isn't really going up, but thread 1 allocates memory, thread 2 allocates, thread 1 frees, but it cannot be coalesced because thread 2</p>



<a name="128599253"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128599253" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128599253">(Jun 25 2018 at 13:14)</a>:</h4>
<p>so the total allocation from the OS gets bigger and bigger, but the actual used doesnt</p>



<a name="128599264"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128599264" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128599264">(Jun 25 2018 at 13:14)</a>:</h4>
<p>heh, I was going to suggest trying a different allocator, do you see this with both jemalloc and system?</p>



<a name="128599274"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128599274" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128599274">(Jun 25 2018 at 13:15)</a>:</h4>
<p>There's only five (one for each Jenkins instance) and then however many hyper/reqwest uses. That stays constant.</p>



<a name="128599277"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128599277" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128599277">(Jun 25 2018 at 13:15)</a>:</h4>
<p>Happens with system and jemalloc.</p>



<a name="128599286"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128599286" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128599286">(Jun 25 2018 at 13:15)</a>:</h4>
<p>I was just using system because I read jemalloc doesn't like valgrind.</p>



<a name="128599287"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128599287" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128599287">(Jun 25 2018 at 13:15)</a>:</h4>
<p>it is true</p>



<a name="128599350"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128599350" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128599350">(Jun 25 2018 at 13:16)</a>:</h4>
<p>If it were memory fragmentation, how would I go about resolving that?</p>



<a name="128599369"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128599369" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128599369">(Jun 25 2018 at 13:17)</a>:</h4>
<p>Been banging my head against a wall for a good week looking through the stacktrace for the allocations looking for something dodgy.</p>



<a name="128599572"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128599572" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128599572">(Jun 25 2018 at 13:21)</a>:</h4>
<p>I honestly don't know. One answer is "use a compacting garbage collector" <span class="emoji emoji-1f622" title="cry">:cry:</span></p>



<a name="128599615"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128599615" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128599615">(Jun 25 2018 at 13:22)</a>:</h4>
<p>I don't suppose you can make it even smaller repro? Using just serde in threads?</p>



<a name="128599644"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128599644" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128599644">(Jun 25 2018 at 13:22)</a>:</h4>
<p>I'll see what I can do about getting the repro smaller.</p>



<a name="128599646"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128599646" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128599646">(Jun 25 2018 at 13:22)</a>:</h4>
<p>Thanks.</p>



<a name="128599655"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128599655" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128599655">(Jun 25 2018 at 13:23)</a>:</h4>
<p>I don't even know how to validate that it is fragmentation</p>



<a name="128599738"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128599738" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128599738">(Jun 25 2018 at 13:24)</a>:</h4>
<blockquote>
<p>causes the memory to keep increasing.</p>
</blockquote>
<p>How are you seeing this manifest? Are you getting errors of some kind?</p>



<a name="128599779"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128599779" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128599779">(Jun 25 2018 at 13:25)</a>:</h4>
<p>On machines with low memory, it would get killed. On my own working machine I can just watch htop or some other utility remark that the process' memory is increasing gradually. It can reach around 40% or so of the system memory (~7 GB) and it increases at around 20MB/s.</p>



<a name="128599854"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128599854" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128599854">(Jun 25 2018 at 13:27)</a>:</h4>
<p>That's a pretty quick rate</p>



<a name="128599874"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128599874" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128599874">(Jun 25 2018 at 13:27)</a>:</h4>
<p><a href="https://github.com/mockersf/jenkins-api.rs/blob/ba7e33540cd159c20d13bab4311406e4a2d31293/src/job/common.rs#L71-L83" target="_blank" title="https://github.com/mockersf/jenkins-api.rs/blob/ba7e33540cd159c20d13bab4311406e4a2d31293/src/job/common.rs#L71-L83">https://github.com/mockersf/jenkins-api.rs/blob/ba7e33540cd159c20d13bab4311406e4a2d31293/src/job/common.rs#L71-L83</a></p>
<p>I see two relatively interesting features there — the <code>flatten</code> and the <code>PhantomData</code></p>



<a name="128599877"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128599877" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128599877">(Jun 25 2018 at 13:27)</a>:</h4>
<p>Yeah, luckily it often terminates before that but on hosts with less memory it won't. With a single thread it would use around 0.1-0.5% of system memory.</p>



<a name="128600158"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128600158" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128600158">(Jun 25 2018 at 13:33)</a>:</h4>
<p><code>_IMPL_DESERIALIZE_FOR_CommonAction</code> , <code>_IMPL_DESERIALIZE_FOR_ShortView</code>, <code>_IMPL_DESERIALIZE_FOR_ShortJob</code>,  all make use of <code>flatten</code></p>



<a name="128600206"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128600206" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128600206">(Jun 25 2018 at 13:34)</a>:</h4>
<p>I know that's a <em>newer</em> feature of Serde, but I cannot think how it would have threading differences at all</p>



<a name="128600236"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128600236" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128600236">(Jun 25 2018 at 13:35)</a>:</h4>
<p>This may be inconsequential, but <a href="https://github.com/sile/libflate/blob/master/src/non_blocking/deflate/decode.rs#L204-L225" target="_blank" title="https://github.com/sile/libflate/blob/master/src/non_blocking/deflate/decode.rs#L204-L225">this code</a> came up when I was running massif the first few times, but on recent runs it hasn't shown up, not sure why. It struck me as potentially the issue, though unlikely.</p>



<a name="128601032"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128601032" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128601032">(Jun 25 2018 at 13:53)</a>:</h4>
<p>I've tried pasting (and anonymizing) a dump from the API and just doing a loop where I parse that a bunch into that <code>CommonJob</code> struct and that doesn't seem to reproduce it.</p>



<a name="128601106"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128601106" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128601106">(Jun 25 2018 at 13:55)</a>:</h4>
<p>That makes me think it's something the API wrapper is doing.</p>



<a name="128601565"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128601565" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128601565">(Jun 25 2018 at 14:05)</a>:</h4>
<p>You can try cloning the jenkins crate locally and adding drop impls which print 'dropping' to the struct you are deserializing as a spot-check, but that does seem unlikely to indicate problems...</p>



<a name="128601584"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128601584" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128601584">(Jun 25 2018 at 14:05)</a>:</h4>
<p>I've thrown in <code>mem::drop(..)</code> manually and I can't see anywhere that they override or mess with the default <code>Drop</code>.</p>



<a name="128601607"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128601607" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128601607">(Jun 25 2018 at 14:06)</a>:</h4>
<div class="codehilite"><pre><span></span>n1: 2799128 0x15E065: serde_json::de::from_reader (de.rs:2178)
 n1: 2799128 0x1A3CC8: reqwest::response::Response::json (response.rs:167)
  n1: 2799128 0x1908E8: jenkins_api::home::&lt;impl jenkins_api::client::Jenkins&gt;::get_home (home.rs:51)
   n1: 2799128 0x132A4A: poller_repro::buggy_code (main.rs:72)
    n1: 2799128 0x133977: poller_repro::run_threads::{{closure}} (dfa.rs:650)
</pre></div>


<p>There's not a lot of "API" around the serde code though</p>



<a name="128601659"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128601659" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128601659">(Jun 25 2018 at 14:07)</a>:</h4>
<p>Instead of printing, you may want like an atomic variable that counts creations and one that counts drops, then print that when it should be equal</p>



<a name="128601667"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128601667" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128601667">(Jun 25 2018 at 14:07)</a>:</h4>
<p>That specifically isn't the big allocation, the call to <code>get_home</code> is fine. The massif data lists allocation backtraces by time, so if you go to the bottom of that dump, then you'll see the big allocation - at the start that allocation may be biggest but it doesn't grow and quickly gets overtaken.</p>



<a name="128601674"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128601674" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128601674">(Jun 25 2018 at 14:07)</a>:</h4>
<p>valgrind's memcheck might tell you where you're missing memory, too</p>



<a name="128601682"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128601682" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128601682">(Jun 25 2018 at 14:07)</a>:</h4>
<p>It prints out "leaks"</p>



<a name="128601732"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128601732" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128601732">(Jun 25 2018 at 14:08)</a>:</h4>
<p>It's the <code>get_full_job</code> call that seems to allocate a lot. I've ran heaptrack on it and observed that its lots of small allocations rather than one large one.</p>



<a name="128601741"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128601741" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128601741">(Jun 25 2018 at 14:08)</a>:</h4>
<p>I've not yet run normal valgrind on the repro code, I'll do that.</p>



<a name="128601744"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128601744" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128601744">(Jun 25 2018 at 14:08)</a>:</h4>
<p>The last thing in the file also doesn't have much:</p>
<div class="codehilite"><pre><span></span>n1: 11111824 0x13B4B5: serde_json::de::from_reader (de.rs:2178)
 n1: 11111824 0x128528: reqwest::response::Response::json (response.rs:167)
  n1: 11111824 0x151D72: &lt;jenkins_api::job::common::ShortJob&lt;T&gt;&gt;::get_full_job (common.rs:93)
   n1: 11111824 0x132F9E: poller_repro::buggy_code2 (main.rs:81)
    n1: 11111824 0x132E38: poller_repro::buggy_code (dfa.rs:1417)
</pre></div>



<a name="128601779"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128601779" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128601779">(Jun 25 2018 at 14:10)</a>:</h4>
<p>I reinstalled my OS recently, so now I'm reinstalling Xcode to get those profiling tools</p>



<a name="128601827"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128601827" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128601827">(Jun 25 2018 at 14:10)</a>:</h4>
<p>just to see if I can do anything</p>



<a name="128601833"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128601833" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128601833">(Jun 25 2018 at 14:10)</a>:</h4>
<p>I recommend using massif_visualizer or ms_print to read it rather than just looking at the raw dump.</p>



<a name="128601869"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128601869" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128601869">(Jun 25 2018 at 14:12)</a>:</h4>
<p>I've updated the gist with some public Jenkins servers (this doesn't poll too hard, so it's probably alright to run it for a minute or two to see the memory tick up, sorry Ubuntu people!) that way you'll be able to see it happening.</p>



<a name="128601936"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128601936" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128601936">(Jun 25 2018 at 14:13)</a>:</h4>
<p>A very small scale DDOS, nice</p>



<a name="128602972"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128602972" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128602972">(Jun 25 2018 at 14:35)</a>:</h4>
<p>I've added Valgrind data, I had to CTRL+C because it would have taken a few hours on my local test Jenkii with the Valgrind performance hit, but you can probably see the largest allocation at the bottom as the one that is problematic.</p>



<a name="128604267"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128604267" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128604267">(Jun 25 2018 at 15:03)</a>:</h4>
<p><span class="user-mention" data-user-id="116107">@David Wood</span> this does exit normally after a while , yah?</p>



<a name="128604297"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128604297" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128604297">(Jun 25 2018 at 15:03)</a>:</h4>
<p>Once it has looped over every job, there's a <em>lot</em> on the Ubuntu jenkins so it will take a while. Normally a minute or so is enough to demonstrate the effect in htop/etc.</p>



<a name="128604349"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128604349" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128604349">(Jun 25 2018 at 15:04)</a>:</h4>
<p><a href="/user_uploads/4715/JWZH9GTV3IzNGRs-x66CBr8e/pasted_image.png" target="_blank" title="pasted_image.png">pasted image</a></p>
<div class="message_inline_image"><a href="/user_uploads/4715/JWZH9GTV3IzNGRs-x66CBr8e/pasted_image.png" target="_blank" title="pasted image"><img src="/user_uploads/4715/JWZH9GTV3IzNGRs-x66CBr8e/pasted_image.png"></a></div>



<a name="128604357"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128604357" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128604357">(Jun 25 2018 at 15:04)</a>:</h4>
<p>It will go back down once some threads finish.</p>



<a name="128604360"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128604360" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128604360">(Jun 25 2018 at 15:04)</a>:</h4>
<p>no leaks <em>reported</em></p>



<a name="128604394"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128604394" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128604394">(Jun 25 2018 at 15:05)</a>:</h4>
<p>That peak is ~750MB</p>



<a name="128604445"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128604445" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128604445">(Jun 25 2018 at 15:06)</a>:</h4>
<p>Do you see the memory just keep increasing when it is running? I don't know if it leaks memory necessarily. But if you run it with the <code>run_no_threads</code>, it hovers around 0.1-0.5% system usage on my machine, whereas with <code>run_threads</code> it will go to 20+, 40+% eventually (though sometimes less too).</p>



<a name="128604497"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128604497" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128604497">(Jun 25 2018 at 15:07)</a>:</h4>
<p><a href="/user_uploads/4715/l0HAI8bpn7yBLc0L5x-fduN4/pasted_image.png" target="_blank" title="pasted_image.png">pasted image</a></p>
<div class="message_inline_image"><a href="/user_uploads/4715/l0HAI8bpn7yBLc0L5x-fduN4/pasted_image.png" target="_blank" title="pasted image"><img src="/user_uploads/4715/l0HAI8bpn7yBLc0L5x-fduN4/pasted_image.png"></a></div>



<a name="128604601"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128604601" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128604601">(Jun 25 2018 at 15:09)</a>:</h4>
<p>That was the whole run for ubuntu repos it went up was stable, went down</p>



<a name="128604695"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128604695" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128604695">(Jun 25 2018 at 15:11)</a>:</h4>
<p>I mean, I guess it could be related to the specific Jenkii I'm testing with. But it seems odd that the memory usage just keeps going up for me when running with multiple threads - I'd imagine there'd be more memory used, sure, but it would be a constant factor on the single-threaded version.</p>



<a name="128604751"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128604751" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128604751">(Jun 25 2018 at 15:12)</a>:</h4>
<p>did you graph your ubuntu version?</p>



<a name="128604865"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128604865" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128604865">(Jun 25 2018 at 15:14)</a>:</h4>
<p><span class="user-mention" data-user-id="116107">@David Wood</span> It's worth pointing out that it looks like the memory is in the Error since at least for me I don't see the println in the Ok at all</p>



<a name="128604867"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128604867" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128604867">(Jun 25 2018 at 15:14)</a>:</h4>
<p>I haven't with the ubuntu repos since I've got local ones to test with. I'll kick that off now. Unless my Jenkii are returning some strange malformed stuff that is causing the memory to just keep increasing..? Seems strange.</p>



<a name="128604891"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128604891" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128604891">(Jun 25 2018 at 15:15)</a>:</h4>
<p>Specifically I don't see <a href="https://gist.github.com/davidtwco/46b9a212654c9b8d8061df056467a20d#file-main-rs-L77" target="_blank" title="https://gist.github.com/davidtwco/46b9a212654c9b8d8061df056467a20d#file-main-rs-L77">https://gist.github.com/davidtwco/46b9a212654c9b8d8061df056467a20d#file-main-rs-L77</a> at all with the gist as is</p>



<a name="128604909"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128604909" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128604909">(Jun 25 2018 at 15:16)</a>:</h4>
<div class="codehilite"><pre><span></span>thread &#39;ubuntu3&#39; panicked at &#39;called `Result::unwrap()` on an `Err` value: Error { kind: Json(Error(&quot;Unexpected eof during chunk size line&quot;, line: 1, column: 19333120)), url: None }&#39;, libcore/result.rs:945:5
note: Run with `RUST_BACKTRACE=1` for a backtrace.
thread &#39;ubuntu2&#39; panicked at &#39;called `Result::unwrap()` on an `Err` value: Error { kind: Json(Error(&quot;Unexpected eof during chunk size line&quot;, line: 1, column: 23142400)), url: None }&#39;, libcore/result.rs:945:5
thread &#39;ubuntu1&#39; panicked at &#39;called `Result::unwrap()` on an `Err` value: Error { kind: Json(Error(&quot;Unexpected eof during chunk size line&quot;, line: 1, column: 26378240)), url: None }&#39;, libcore/result.rs:945:5
</pre></div>



<a name="128604953"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128604953" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128604953">(Jun 25 2018 at 15:16)</a>:</h4>
<p>I didn't make it particularly robust, but I see job names being printed regularly.</p>



<a name="128605053"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128605053" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128605053">(Jun 25 2018 at 15:18)</a>:</h4>
<p><span class="user-mention" data-user-id="116107">@David Wood</span> It looks to me like the main allocations that keep growing come from the jenkins.get_home() or something -- not from get_full_job</p>



<a name="128605081"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128605081" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128605081">(Jun 25 2018 at 15:18)</a>:</h4>
<p>It's possible, I only say <code>get_full_job</code> because of all the various massif and heaptrack runs I've done, I always see <code>get_full_job</code> as the largest contributor.</p>



<a name="128605085"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128605085" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128605085">(Jun 25 2018 at 15:19)</a>:</h4>
<p><span class="user-mention" data-user-id="116107">@David Wood</span>  Are you running in dev or release?</p>



<a name="128605101"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128605101" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128605101">(Jun 25 2018 at 15:19)</a>:</h4>
<p>(I'm using a debug build)</p>



<a name="128605103"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128605103" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128605103">(Jun 25 2018 at 15:19)</a>:</h4>
<p>Currently dev, but I've seen it in release too.</p>



<a name="128605130"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128605130" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128605130">(Jun 25 2018 at 15:19)</a>:</h4>
<p>I originally didn't notice it in the normal application because I ran it in short bursts to test - it was only when I deployed it with a release build did I see it get killed and started investigating.</p>



<a name="128605328"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128605328" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128605328">(Jun 25 2018 at 15:23)</a>:</h4>
<p><span class="user-mention" data-user-id="116122">@simulacrum</span> On the Ubuntu Jenkins, all I get is <code>Err(e)</code> too. But on my own Jenkins instances, I see <code>Ok(..)</code>.</p>



<a name="128605329"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128605329" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128605329">(Jun 25 2018 at 15:23)</a>:</h4>
<p>Hadn't noticed that before.</p>



<a name="128605334"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128605334" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128605334">(Jun 25 2018 at 15:23)</a>:</h4>
<p>I still see the steady increase though.</p>



<a name="128605410"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128605410" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128605410">(Jun 25 2018 at 15:24)</a>:</h4>
<p>It might be that the increase is not as dramatic if the data returned from the API calls is smaller - on my own Jenkins instances, they return quite a lot of data, probably more than the individual ubuntu instance API calls are returning - probably why I see more memory used.</p>



<a name="128605418"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128605418" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128605418">(Jun 25 2018 at 15:24)</a>:</h4>
<p>Though the same increasing effect is observed.</p>



<a name="128605498"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128605498" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128605498">(Jun 25 2018 at 15:26)</a>:</h4>
<p>Have you seen that the no threads function does not have a steady increase but stays pretty much the same? That's mostly where I'm confused - if that increased gradually, then at least it wouldn't be the threads. But it's that when introducing threads - that don't interact - suddenly there's an increase rather than a constant factor more memory usage.</p>



<a name="128605623"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128605623" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128605623">(Jun 25 2018 at 15:29)</a>:</h4>
<p>Let me look</p>



<a name="128605693"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128605693" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128605693">(Jun 25 2018 at 15:30)</a>:</h4>
<p><a href="/user_uploads/4715/Hdi3nTfpV73-cps5ElmpfZJV/pasted_image.png" target="_blank" title="pasted_image.png">pasted image</a>  <br>
my graph for the ubuntu ones - there's still a increase before it crashes, though not as sharp as I think it is returning less information in the API calls.</p>
<div class="message_inline_image"><a href="/user_uploads/4715/Hdi3nTfpV73-cps5ElmpfZJV/pasted_image.png" target="_blank" title="pasted image"><img src="/user_uploads/4715/Hdi3nTfpV73-cps5ElmpfZJV/pasted_image.png"></a></div>



<a name="128605726"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128605726" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128605726">(Jun 25 2018 at 15:30)</a>:</h4>
<p>How big is your <code>instances</code>? e.g. how many threads?</p>



<a name="128605732"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128605732" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128605732">(Jun 25 2018 at 15:31)</a>:</h4>
<p>Yeah - I don't think there's a memory leak, it is freed when the thread terminates at the end (if allowed to finish naturally), but I'd expect it to be freed before that (and it seemingly is when threads are not involved).</p>



<a name="128605738"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128605738" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128605738">(Jun 25 2018 at 15:31)</a>:</h4>
<p>Same as the example I gave for ubuntu, five.</p>



<a name="128605744"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128605744" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128605744">(Jun 25 2018 at 15:31)</a>:</h4>
<p>Though I have reproduced it with less (I forget exactly how few I've tried - I was checking whether any one of my Jenkins instances was causing the issue but alas, no)</p>



<a name="128605818"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128605818" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128605818">(Jun 25 2018 at 15:32)</a>:</h4>
<p>Is it possible that while the majority of the <em>allocations</em> come from the get_full_job call the majority of the memory use is the jobs list (which lives until thread end)</p>



<a name="128605860"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128605860" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128605860">(Jun 25 2018 at 15:33)</a>:</h4>
<p>I had considered that, but looking through the source of <code>jenkins-api</code>, <code>get_full_job</code> just takes the url of the job (a <code>String</code>)  from the <code>ShortJob</code> in <code>home.jobs</code> and does a separate query on that (or slightly modified) URL - returning a new struct that we own. I'd understand it if it was a reference that had the same lifetime as the Jenkins instance, or as the <code>ShortJob</code>, but it isn't?</p>



<a name="128605920"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128605920" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128605920">(Jun 25 2018 at 15:34)</a>:</h4>
<p>Yeah, I think you can see it with just one thread (that isn't main thread)</p>



<a name="128605934"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128605934" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128605934">(Jun 25 2018 at 15:34)</a>:</h4>
<p>And even if that were the case, why wouldn't we see an, albeit smaller, but steady increase with the one thread?</p>



<a name="128605935"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128605935" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128605935">(Jun 25 2018 at 15:34)</a>:</h4>
<p>Yes but <code>ShortJob</code> seems to be a fairly large struct in theory -- it has that #[serde(flatten)] other field...</p>
<p>In other words, <code>ShortJob</code> lives until thread end</p>



<a name="128605953"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128605953" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128605953">(Jun 25 2018 at 15:35)</a>:</h4>
<p>Ah, yeah, good point.</p>



<a name="128605974"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128605974" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128605974">(Jun 25 2018 at 15:35)</a>:</h4>
<p>So maybe we're basically seeing the same thing it's just less noticeable in the non-threaded case...</p>



<a name="128606030"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128606030" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128606030">(Jun 25 2018 at 15:36)</a>:</h4>
<blockquote>
<p>Yeah, I think you can see it with just one thread (that isn't main thread)</p>
</blockquote>
<p>Is this referring to the how many instances query?</p>



<a name="128606083"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128606083" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128606083">(Jun 25 2018 at 15:37)</a>:</h4>
<p>It's also possibly worth noting that it looks like these responses are fairly huge -- &gt;45 MB for the one that crashes at least... and serde is likely storing effectively the whole response in memory via flatten etc</p>



<a name="128606095"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128606095" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128606095">(Jun 25 2018 at 15:37)</a>:</h4>
<blockquote>
<p>So maybe we're basically seeing the same thing it's just less noticeable in the non-threaded case...</p>
</blockquote>
<p>Perhaps. Looking at a API response from Jenkins (just prefix a URL of a job with <code>/api/json</code>), the shortjob normally is only like three fields.</p>



<a name="128606109"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128606109" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128606109">(Jun 25 2018 at 15:37)</a>:</h4>
<p>hm, okay</p>



<a name="128606175"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128606175" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128606175">(Jun 25 2018 at 15:38)</a>:</h4>
<p>Here's <a href="https://jenkins.qa.ubuntu.com/view/All/job/account-plugins-utopic-amd64-ci/api/json" target="_blank" title="https://jenkins.qa.ubuntu.com/view/All/job/account-plugins-utopic-amd64-ci/api/json">an example</a>. It has two builds whereas in my internal Jenkins instances, I have potentially hundreds, so the response is huge.</p>



<a name="128606208"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128606208" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128606208">(Jun 25 2018 at 15:39)</a>:</h4>
<p>Threaded case for me:</p>
<div class="codehilite"><pre><span></span>GB
1.760^                                                          #
     |                                                       @@@#         :@:
     |                                                    @@@@@@#      ::::@:
     |                                                 @@@@@@@@@#:  ::@::::@:
     |                                             ::::@@@@@@@@@#:::::@::::@:
     |                                            :::: @@@@@@@@@#:::::@::::@:
     |                                        :@:::::: @@@@@@@@@#:::::@::::@:
     |                                     @:@:@: :::: @@@@@@@@@#:::::@::::@:
     |                                  ::@@:@:@: :::: @@@@@@@@@#:::::@::::@:
     |                               ::@::@@:@:@: :::: @@@@@@@@@#:::::@::::@:
     |                             @@: @::@@:@:@: :::: @@@@@@@@@#:::::@::::@::
     |                         :@@@@@: @::@@:@:@: :::: @@@@@@@@@#:::::@::::@::
     |                       @@:@@ @@: @::@@:@:@: :::: @@@@@@@@@#:::::@::::@::
     |                    @@@@@:@@ @@: @::@@:@:@: :::: @@@@@@@@@#:::::@::::@::
     |                 @@@@ @@@:@@ @@: @::@@:@:@: :::: @@@@@@@@@#:::::@::::@::
     |              @@@@@@@ @@@:@@ @@: @::@@:@:@: :::: @@@@@@@@@#:::::@::::@::
     |           @@:@@ @@@@ @@@:@@ @@: @::@@:@:@: :::: @@@@@@@@@#:::::@::::@::
     |        :@@@ :@@ @@@@ @@@:@@ @@: @::@@:@:@: :::: @@@@@@@@@#:::::@::::@::
     |     :@@@@@@ :@@ @@@@ @@@:@@ @@: @::@@:@:@: :::: @@@@@@@@@#:::::@::::@::
     |  @@@:@ @@@@ :@@ @@@@ @@@:@@ @@: @::@@:@:@: :::: @@@@@@@@@#:::::@::::@::
   0 +-----------------------------------------------------------------------&gt;Gi
     0                                                                   68.91
</pre></div>


<p>Single:</p>
<div class="codehilite"><pre><span></span>    MB
458.3^              #
     |              #                                          :             :
     |              #                           :             ::             @
     |            @@#            :@            ::             ::            :@
     |            @ #           ::@           :::            :::           ::@
     |           :@ #          :::@           :::           ::::          :::@
     |           :@ #          :::@          ::::           ::::          :::@
     |         :::@ #         ::::@         :::::         ::::::         ::::@
     |         : :@ #        :::::@         :::::         : ::::        :::::@
     |        @: :@ #        :::::@       :::::::        :: ::::       ::::::@
     |        @: :@ #      :::::::@       : :::::       ::: ::::      :::::::@
     |      @@@: :@ #      : :::::@      :: :::::      :::: ::::      :::::::@
     |      @ @: :@ #     @: :::::@     @:: :::::      :::: ::::      :::::::@
     |     @@ @: :@ #     @: :::::@     @:: :::::     ::::: ::::     @:::::::@
     |    @@@ @: :@ #   ::@: :::::@    :@:: :::::     ::::: ::::    :@:::::::@
     |    @@@ @: :@ #   : @: :::::@    :@:: :::::   ::::::: ::::   ::@:::::::@
     |  @@@@@ @: :@ #  :: @: :::::@   ::@:: :::::   : ::::: ::::   ::@:::::::@
     |  @ @@@ @: :@ #  :: @: :::::@   ::@:: :::::   : ::::: ::::  :::@:::::::@
     | @@ @@@ @: :@ # ::: @: :::::@ ::::@:: :::::  :: ::::: :::: ::::@:::::::@
     | @@ @@@ @: :@ #:::: @: :::::@:: ::@:: :::::  :: ::::: :::: ::::@:::::::@
   0 +-----------------------------------------------------------------------&gt;Gi
     0                                                                   71.92
</pre></div>



<a name="128606218"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128606218" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128606218">(Jun 25 2018 at 15:39)</a>:</h4>
<p>Each entry in the <code>build</code> list from the response being a <code>ShortJob</code>. It takes the url from it, appends <code>/api/json</code> and parses that into a <code>CommonJob</code> in <code>get_full_job</code>.</p>



<a name="128606296"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128606296" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128606296">(Jun 25 2018 at 15:40)</a>:</h4>
<p>I think that is what I would expect, in the single threaded case, it frees each <code>CommonJob</code> and then fetches another. In the multi-threaded case, it just keeps growing (as if those <code>CommonJob</code>s, allocated by <code>get_full_job</code> are not being freed).</p>



<a name="128606308"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128606308" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128606308">(Jun 25 2018 at 15:40)</a>:</h4>
<p>We're freeing common jobs -- just not short jobs</p>



<a name="128606319"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128606319" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128606319">(Jun 25 2018 at 15:41)</a>:</h4>
<p>or at least, we should be</p>



<a name="128606483"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128606483" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128606483">(Jun 25 2018 at 15:43)</a>:</h4>
<p>Yeah, that's what I'd expect. But we only fetch the <code>ShortJob</code>s once - in the <code>get_home</code> call - so it shouldn't be increasing over time. (I realise that the API example I gave previously was slightly incorrect, as we get the short job from the home query and then the <code>CommonJob</code> from the job page query - the builds key we were looking at is another thing entirely).</p>



<a name="128606512"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128606512" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128606512">(Jun 25 2018 at 15:43)</a>:</h4>
<p>Only the <code>CommonJob</code>s are fetched over time, and therefore could contribute to the increasing memory (at least as far as I can work out).</p>



<a name="128606561"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128606561" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128606561">(Jun 25 2018 at 15:44)</a>:</h4>
<p>I agree that what I would expect is that as we fetch <code>ShortJob</code>s we should increase (faster in the threaded case) and then the commonjobs should be a up/down (effectively average flat line)</p>



<a name="128606583"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128606583" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128606583">(Jun 25 2018 at 15:44)</a>:</h4>
<p>Yeah, and the <code>ShortJob</code>s should live the duration of the execution.</p>



<a name="128606624"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128606624" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128606624">(Jun 25 2018 at 15:45)</a>:</h4>
<p>(This is where we get the <code>ShortJob</code> from in the API - <a href="https://jenkins.qa.ubuntu.com//api/json" target="_blank" title="https://jenkins.qa.ubuntu.com//api/json">link</a> (ala <code>get_home</code>) - and then something like this - <a href="https://jenkins.qa.ubuntu.com/view/All/job/account-plugins-utopic-amd64-ci/api/json" target="_blank" title="https://jenkins.qa.ubuntu.com/view/All/job/account-plugins-utopic-amd64-ci/api/json">link</a> (ala <code>get_full_job</code>) - is a <code>CommonJob</code> - I got this mixed up earlier).</p>



<a name="128606701"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128606701" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128606701">(Jun 25 2018 at 15:46)</a>:</h4>
<p>CommonJobs do look fairly small</p>



<a name="128606707"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128606707" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128606707">(Jun 25 2018 at 15:46)</a>:</h4>
<p>I'd expect a increase in memory usage at the start to a sort-of base-level as all the <code>ShortJob</code> are loaded, then up and down spikes as <code>CommonJob</code> is loaded and unloaded as the iteration proceeds.</p>



<a name="128606745"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128606745" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128606745">(Jun 25 2018 at 15:47)</a>:</h4>
<p>That reponse is slightly less as it doesn't use <code>?depth=1</code> like the example code (just returns more info - don't think it makes a difference in terms of this behaviour we're seeing) and the job I linked has very few builds.</p>



<a name="128606802"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128606802" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128606802">(Jun 25 2018 at 15:48)</a>:</h4>
<p>I apologize if I'm doing a awful job at explaining and clarifying what I mean.</p>



<a name="128606830"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128606830" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128606830">(Jun 25 2018 at 15:49)</a>:</h4>
<p>massif is reporting that ~60% of the allocated bytes come from the home call</p>



<a name="128606856"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128606856" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128606856">(Jun 25 2018 at 15:49)</a>:</h4>
<p>Or at least that's how I understand the output</p>



<a name="128606916"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128606916" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128606916">(Jun 25 2018 at 15:50)</a>:</h4>
<p>So maybe in both cases we're seeing the same behavior effectively -- steady growth, and then the get_full_job calls are fast and as such almost unnoticeable for ms_print... let me try switching it to time-based graph</p>



<a name="128606951"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128606951" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128606951">(Jun 25 2018 at 15:51)</a>:</h4>
<p>Are you using massif visualizer to view the data in a nice way?</p>



<a name="128606953"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128606953" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128606953">(Jun 25 2018 at 15:51)</a>:</h4>
<p>ms_print</p>



<a name="128607007"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128607007" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128607007">(Jun 25 2018 at 15:52)</a>:</h4>
<p>Here's a link to some other massif data I've collected: <a href="https://www.dropbox.com/sh/5i0vxq19v5tr4wy/AAAuZHS2atedrv6eDgCxx_8oa?dl=0" target="_blank" title="https://www.dropbox.com/sh/5i0vxq19v5tr4wy/AAAuZHS2atedrv6eDgCxx_8oa?dl=0">link</a> with my own Jenkins instances.</p>
<div class="message_inline_ref"><a href="https://www.dropbox.com/sh/5i0vxq19v5tr4wy/AAAuZHS2atedrv6eDgCxx_8oa?dl=0" target="_blank" title="Temporary Jenkins Memory Profiling"><img src="https://www.dropbox.com/static/images/spectrum-icons/generated/content/content-folder_dropbox-large.png"></a><div><div class="message_inline_image_title">Temporary Jenkins Memory Profiling</div><desc class="message_inline_image_desc"></desc></div></div>



<a name="128607533"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128607533" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128607533">(Jun 25 2018 at 16:00)</a>:</h4>
<blockquote>
<p>line: 1, column: 35758080</p>
</blockquote>
<p>35,758,080 that's 35MB</p>



<a name="128607544"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128607544" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128607544">(Jun 25 2018 at 16:00)</a>:</h4>
<p>What's that from?</p>



<a name="128607546"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128607546" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128607546">(Jun 25 2018 at 16:00)</a>:</h4>
<p>that seems odd, are any of these responses supposed to be that big</p>



<a name="128607558"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128607558" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128607558">(Jun 25 2018 at 16:01)</a>:</h4>
<blockquote>
<p>thread 'ubuntu2' panicked at 'called <code>Result::unwrap()</code> on an <code>Err</code> value: Error { kind: Json(Error("Unexpected eof during chunk size line", line: 1, column: 35758080)), url: None }', libcore/result.rs:945:5</p>
</blockquote>



<a name="128607561"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128607561" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128607561">(Jun 25 2018 at 16:01)</a>:</h4>
<p>Yeah, not sure. There doesn't seem to be a difference between the two profiles really, as far as I can tell</p>



<a name="128607566"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128607566" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128607566">(Jun 25 2018 at 16:01)</a>:</h4>
<p>Yeah, they certainly could be. Jenkins can return <em>lots</em> of data.</p>



<a name="128607570"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128607570" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128607570">(Jun 25 2018 at 16:01)</a>:</h4>
<p>jeez jenkins</p>



<a name="128607580"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128607580" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128607580">(Jun 25 2018 at 16:01)</a>:</h4>
<p>I have jobs that just time out because we can't get the data fast enough.</p>



<a name="128607625"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128607625" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128607625">(Jun 25 2018 at 16:02)</a>:</h4>
<p>Do either of you agree that there is some strange behaviour here or am I just going mad?</p>



<a name="128607639"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128607639" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128607639">(Jun 25 2018 at 16:02)</a>:</h4>
<p>RIIR</p>



<a name="128607676"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128607676" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128607676">(Jun 25 2018 at 16:03)</a>:</h4>
<p>If the common jobs are actually large, then yes, but if they're very small, then the results I get are expected</p>



<a name="128607740"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128607740" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128607740">(Jun 25 2018 at 16:04)</a>:</h4>
<p>How is it explained if they are small?</p>



<a name="128607766"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128607766" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128607766">(Jun 25 2018 at 16:04)</a>:</h4>
<p>Well, mostly explained -- we see growth only as the get_home call runs and then growth "stops" in theory if I'm right about where this comes from</p>



<a name="128607884"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128607884" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128607884">(Jun 25 2018 at 16:07)</a>:</h4>
<p>I see the get_home call complete in ~2-4 seconds and then the iteration through it take &lt;1 second</p>



<a name="128607887"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128607887" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128607887">(Jun 25 2018 at 16:07)</a>:</h4>
<p><a href="/user_uploads/4715/Ncf4JPiyzPr9Ps4csnlWR6Hj/pasted_image.png" target="_blank" title="pasted_image.png">pasted image</a>  Here's an example of a large <code>CommonJob</code> response I picked at random from my own instances.</p>
<div class="message_inline_image"><a href="/user_uploads/4715/Ncf4JPiyzPr9Ps4csnlWR6Hj/pasted_image.png" target="_blank" title="pasted image"><img src="/user_uploads/4715/Ncf4JPiyzPr9Ps4csnlWR6Hj/pasted_image.png"></a></div>



<a name="128607984"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128607984" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128607984">(Jun 25 2018 at 16:08)</a>:</h4>
<p>i.e., sure, we have large CommonJobs but they're nearly immediately deallocated</p>



<a name="128607996"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128607996" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128607996">(Jun 25 2018 at 16:09)</a>:</h4>
<p>If the <code>CommonJob</code> allocations are small (or being correctly deallocated), then do you propose the <code>get_home</code> call is allocating the growing memory over time or something else?</p>



<a name="128608012"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128608012" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128608012">(Jun 25 2018 at 16:09)</a>:</h4>
<p>right, yeah, that's where I think the memory growth comes from</p>



<a name="128608041"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128608041" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128608041">(Jun 25 2018 at 16:10)</a>:</h4>
<p>Wouldn't that only allocate as the <code>get_home</code> call is happening - and afterwards during iteration, be constant?</p>



<a name="128608093"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128608093" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128608093">(Jun 25 2018 at 16:10)</a>:</h4>
<p>right but I see iteration happen instantaneously in a release build at least</p>



<a name="128608113"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128608113" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128608113">(Jun 25 2018 at 16:10)</a>:</h4>
<p>I'm open to the idea, whatever it is that causes this, I'd just want to figure out why it's happening. I've not looked into <code>get_home</code> much because my massif profiles have always pointed at <code>get_full_job</code>.</p>



<a name="128608133"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128608133" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128608133">(Jun 25 2018 at 16:11)</a>:</h4>
<p>I don't see get_full_job in profiles at all...</p>



<a name="128608231"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128608231" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128608231">(Jun 25 2018 at 16:13)</a>:</h4>
<p><a href="/user_uploads/4715/ZC0sGMh9dAaBe7xCv9UIcKEc/pasted_image.png" target="_blank" title="pasted_image.png">pasted image</a> <br>
This is from the massif visualizer for data I've shared (particularly 19141) - if I go to the later snapshots, and dig down into the largest allocation location, I see <code>get_full_job</code>.</p>
<div class="message_inline_image"><a href="/user_uploads/4715/ZC0sGMh9dAaBe7xCv9UIcKEc/pasted_image.png" target="_blank" title="pasted image"><img src="/user_uploads/4715/ZC0sGMh9dAaBe7xCv9UIcKEc/pasted_image.png"></a></div>



<a name="128608250"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128608250" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128608250">(Jun 25 2018 at 16:13)</a>:</h4>
<p>Let me try loading that file</p>



<a name="128608298"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128608298" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128608298">(Jun 25 2018 at 16:14)</a>:</h4>
<p>(the image isn't great because I run massif visualizer in a VM - not supported on Windows - and the VM can't be larger than 1920x1080 and I need to resize the window partially offscreen to fit it all in because there's no horizontal scrollbar)</p>



<a name="128608305"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128608305" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128608305">(Jun 25 2018 at 16:14)</a>:</h4>
<p>I see the same result in all my profiles.</p>



<a name="128608360"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128608360" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128608360">(Jun 25 2018 at 16:14)</a>:</h4>
<p>The massif data lists the backtraces of memory allocations, ordered by % of total allocated memory which came from that location, for each snapshot in time, the most recent snapshot at the bottom.</p>



<a name="128608440"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128608440" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128608440">(Jun 25 2018 at 16:15)</a>:</h4>
<p>So it ends up being a little bit up from the bottom of the file that you see the big "89% of allocated memory from X" in "snapshot 47" (for example) one that we want.</p>



<a name="128608504"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128608504" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128608504">(Jun 25 2018 at 16:16)</a>:</h4>
<p>hm, yeah, that might be debug vs. release build difference</p>



<a name="128608516"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128608516" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128608516">(Jun 25 2018 at 16:16)</a>:</h4>
<p>let me try a debug build...</p>



<a name="128608601"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128608601" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128608601">(Jun 25 2018 at 16:18)</a>:</h4>
<p>It's interesting that there is a difference - I've not profiled the release builds, only observed the issue on them.</p>



<a name="128608709"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128608709" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128608709">(Jun 25 2018 at 16:20)</a>:</h4>
<p>well, in theory we're not reading anything but display_name from full job which is tiny so LLVM could skip a bunch of allocation in theory</p>



<a name="128608735"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128608735" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128608735">(Jun 25 2018 at 16:21)</a>:</h4>
<p>Ah. In the main application that I encountered this on, I take much more from the struct, so in release builds of that (which is what I've checked), it would still be pretty bad.</p>



<a name="128608744"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128608744" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128608744">(Jun 25 2018 at 16:21)</a>:</h4>
<p>Printing the name was all that was included here as it was sufficient to demonstrate it.</p>



<a name="128608821"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128608821" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128608821">(Jun 25 2018 at 16:22)</a>:</h4>
<p>that could explain it</p>



<a name="128609149"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128609149" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128609149">(Jun 25 2018 at 16:30)</a>:</h4>
<p>Would it be possible to send your massif output <span class="user-mention" data-user-id="116122">@simulacrum</span> so I can see what it looks like? Curious about the increased allocations from <code>get_home</code>.</p>



<a name="128609271"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128609271" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128609271">(Jun 25 2018 at 16:31)</a>:</h4>
<p>Sure, yeah</p>



<a name="128609326"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128609326" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128609326">(Jun 25 2018 at 16:32)</a>:</h4>
<p><a href="/user_uploads/4715/raBzSVaFWm-cgdIdE7jbbs_w/massif.out.single.bytes" target="_blank" title="massif.out.single.bytes">massif.out.single.bytes</a> <a href="/user_uploads/4715/X4aVLafF02wPx52SeYGP1Rhh/massif.out.single" target="_blank" title="massif.out.single">massif.out.single</a> <a href="/user_uploads/4715/U9rx_QTpQwbYAklBsFVvXOgU/massif.out.threaded.bytes" target="_blank" title="massif.out.threaded.bytes">massif.out.threaded.bytes</a></p>



<a name="128609342"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128609342" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128609342">(Jun 25 2018 at 16:32)</a>:</h4>
<p>What's different about the <code>*.bytes</code> profiles?</p>



<a name="128609343"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128609343" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128609343">(Jun 25 2018 at 16:32)</a>:</h4>
<p><a href="/user_uploads/4715/9za0eoHccR1uVQPcTk2MkvNz/massif.out.threaded" target="_blank" title="massif.out.threaded">massif.out.threaded</a> <a href="/user_uploads/4715/-KyHWIVvMl1rUjspw6vI52xW/massif.debug.multi" target="_blank" title="massif.debug.multi">massif.debug.multi</a></p>



<a name="128609351"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128609351" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128609351">(Jun 25 2018 at 16:33)</a>:</h4>
<p>They aren't all that interesting -- <code>--time-unit=B</code> vs. <code>--time-unit=I</code></p>



<a name="128609356"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128609356" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128609356">(Jun 25 2018 at 16:33)</a>:</h4>
<p>Ah, cool.</p>



<a name="128609524"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128609524" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128609524">(Jun 25 2018 at 16:36)</a>:</h4>
<p>That is interesting. They are very different from my own debug profiles.</p>



<a name="128609533"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128609533" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128609533">(Jun 25 2018 at 16:36)</a>:</h4>
<p>all but .debug.multi I think are release profiles</p>



<a name="128609536"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128609536" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128609536">(Jun 25 2018 at 16:36)</a>:</h4>
<p>but you should be able to see that in the profile (based on the command run)</p>



<a name="128609553"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128609553" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128609553">(Jun 25 2018 at 16:37)</a>:</h4>
<p>Yeah. I think in .debug.multi you need <code>--depth=100</code> (or similar) in order to see what call it originates from as the backtrace is much larger on debug builds.</p>



<a name="128609557"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128609557" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128609557">(Jun 25 2018 at 16:37)</a>:</h4>
<p>yeah, possible</p>



<a name="128609619"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128609619" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128609619">(Jun 25 2018 at 16:38)</a>:</h4>
<p>Well, I can say that I'm no closer to understanding this problem. If anything, I've gotten more confused.</p>



<a name="128609635"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128609635" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128609635">(Jun 25 2018 at 16:39)</a>:</h4>
<p>Mostly agreed -- it looks like there's somewhat of a "leak" in that memory use constantly increases but I can't see how that could be thread-related</p>



<a name="128609830"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128609830" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128609830">(Jun 25 2018 at 16:43)</a>:</h4>
<p>Yeah, it's the threading that has been confusing me. I'd expect it to be a problem regardless of threads, or not at all.</p>



<a name="128610053"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128610053" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128610053">(Jun 25 2018 at 16:47)</a>:</h4>
<p>However, I'm not completely certain that I see different behavior</p>



<a name="128610117"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128610117" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128610117">(Jun 25 2018 at 16:48)</a>:</h4>
<p>I think it's the same thing, just manifesting itself differently in the profile - the end result of a gradually increasing memory footprint, only when threaded, is the same.</p>



<a name="128610142"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128610142" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128610142">(Jun 25 2018 at 16:50)</a>:</h4>
<p>My theory, as much as it makes no sense, is that the <code>CommonJob</code>s from <code>get_full_job</code> is where the allocations are accumulating and for some reason, when multi-threaded, are dropped at the end of the thread, and when single-threaded, are dropped at the end of the function. It makes _zero_ sense, but that lines up with what I think I'm observing.</p>



<a name="128610194"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128610194" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128610194">(Jun 25 2018 at 16:50)</a>:</h4>
<p>And with the graphs you posted earlier, if we assume that the <code>get_home</code> allocation is just a constant size that both include.</p>



<a name="128610197"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128610197" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128610197">(Jun 25 2018 at 16:50)</a>:</h4>
<p>I don't think I see a difference threaded vs. not threaded -- both profiles have steady increase for thread lifetime</p>



<a name="128610209"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128610209" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128610209">(Jun 25 2018 at 16:51)</a>:</h4>
<p>i.e. there is no period in them when the bytes allocated is constant</p>



<a name="128610223"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128610223" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128610223">(Jun 25 2018 at 16:51)</a>:</h4>
<p>I would expect that in both, we would see spikes of allocations and de-allocations, but the spikes when multi-threaded would be larger.</p>



<a name="128610228"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128610228" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128610228">(Jun 25 2018 at 16:51)</a>:</h4>
<p>(in the ideal case)</p>



<a name="128610277"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128610277" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128610277">(Jun 25 2018 at 16:52)</a>:</h4>
<p>But then again, I have no idea.</p>



<a name="128610304"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128610304" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128610304">(Jun 25 2018 at 16:53)</a>:</h4>
<p>I'd expect that the thread upon start would show steady growth as get_home runs and then a spikey flat line for the iteration, allocating/deallocating commonjobs</p>



<a name="128610355"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128610355" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128610355">(Jun 25 2018 at 16:54)</a>:</h4>
<p>Yeah, that's what I mean. But instead we see a steady increase instead of the spikes.</p>



<a name="128610362"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128610362" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128610362">(Jun 25 2018 at 16:54)</a>:</h4>
<p>well, instead of the spikey flat line, right?</p>



<a name="128610367"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128610367" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128610367">(Jun 25 2018 at 16:54)</a>:</h4>
<p>Yeah.</p>



<a name="128610380"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128610380" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128610380">(Jun 25 2018 at 16:55)</a>:</h4>
<p>Yeah, I think so -- and we don't know why</p>



<a name="128610390"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128610390" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128610390">(Jun 25 2018 at 16:55)</a>:</h4>
<p>I think we've now agreed on what the problem is.</p>



<a name="128610457"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128610457" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128610457">(Jun 25 2018 at 16:56)</a>:</h4>
<p>Yes, though we're no closer to finding out why :)</p>



<a name="128610482"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128610482" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128610482">(Jun 25 2018 at 16:57)</a>:</h4>
<p>Yes. In fact, I've regressed in my understanding of this issue since I discovered it last week.</p>



<a name="128610923"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128610923" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128610923">(Jun 25 2018 at 17:03)</a>:</h4>
<p>That's usually a good thing, while not feeling like it ;-)</p>



<a name="128611357"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128611357" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128611357">(Jun 25 2018 at 17:13)</a>:</h4>
<p>You know, I wonder if it's a bug with Hyper / Tokio</p>
<p><a href="/user_uploads/4715/hzCG8q0GRVu3Kt5RWFEIo_Dm/pasted_image.png" target="_blank" title="pasted_image.png">pasted image</a> </p>
<div class="message_inline_image"><a href="/user_uploads/4715/hzCG8q0GRVu3Kt5RWFEIo_Dm/pasted_image.png" target="_blank" title="pasted image"><img src="/user_uploads/4715/hzCG8q0GRVu3Kt5RWFEIo_Dm/pasted_image.png"></a></div><p>I added some prints:</p>
<div class="codehilite"><pre><span></span><span class="w">    </span><span class="k">for</span><span class="w"> </span><span class="p">(</span><span class="n">i</span><span class="p">,</span><span class="w"> </span><span class="n">job</span><span class="p">)</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="n">home</span><span class="p">.</span><span class="n">jobs</span><span class="p">.</span><span class="n">into_iter</span><span class="p">().</span><span class="n">enumerate</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">        </span><span class="n">eprintln</span><span class="o">!</span><span class="p">(</span><span class="s">&quot;Mark generation {} now&quot;</span><span class="p">,</span><span class="w"> </span><span class="n">i</span><span class="p">);</span><span class="w"></span>
<span class="w">        </span><span class="k">use</span><span class="w"> </span><span class="n">std</span>::<span class="n">io</span>::<span class="n">Write</span><span class="p">;</span><span class="w"></span>
<span class="w">        </span><span class="n">std</span>::<span class="n">io</span>::<span class="n">stderr</span><span class="p">().</span><span class="n">flush</span><span class="p">().</span><span class="n">unwrap</span><span class="p">();</span><span class="w"></span>
<span class="w">        </span><span class="n">thread</span>::<span class="n">sleep_ms</span><span class="p">(</span><span class="mi">5_000</span><span class="p">);</span><span class="w"></span>
<span class="w">        </span><span class="n">buggy_code2</span><span class="p">(</span><span class="o">&amp;</span><span class="n">jenkins</span><span class="p">,</span><span class="w"> </span><span class="n">job</span><span class="p">);</span><span class="w"></span>
<span class="w">    </span><span class="p">}</span><span class="w"></span>
</pre></div>


<p>but those never showed up. It used 800+ MB without ever running that <code>eprintln</code>...</p>



<a name="128611758"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128611758" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128611758">(Jun 25 2018 at 17:20)</a>:</h4>
<p>If that's from the initial population of home then that would be reasonable...</p>



<a name="128611774"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128611774" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128611774">(Jun 25 2018 at 17:21)</a>:</h4>
<p>What do you mean by initial population? Does home not only get populated once (per thread)?</p>



<a name="128611780"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128611780" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128611780">(Jun 25 2018 at 17:21)</a>:</h4>
<p>(may be confused with terminology)</p>



<a name="128611863"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128611863" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128611863">(Jun 25 2018 at 17:23)</a>:</h4>
<blockquote>
<p>This may be inconsequential, but <a href="https://github.com/sile/libflate/blob/master/src/non_blocking/deflate/decode.rs#L204-L225" target="_blank" title="https://github.com/sile/libflate/blob/master/src/non_blocking/deflate/decode.rs#L204-L225">this code</a> came up when I was running massif the first few times, but on recent runs it hasn't shown up, not sure why. It struck me as potentially the issue, though unlikely.</p>
</blockquote>
<p>I sent this earlier, some other information - when I was originally debugging this particular part of code (when it was still coming up in backtraces, I've not looked since or investigated why it no longer is), I placed a conditional breakpoint in that code that never ever got triggered but the problem went away when I did so IIRC (this was a week ago).</p>



<a name="128611953"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128611953" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128611953">(Jun 25 2018 at 17:25)</a>:</h4>
<blockquote>
<p><code>impl&lt;'a&gt; ToString for Path&lt;'a&gt; {</code></p>
</blockquote>
<p>That's a PR waiting to happen</p>



<a name="128611954"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128611954" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128611954">(Jun 25 2018 at 17:25)</a>:</h4>
<p>Yes, that initial and only population</p>



<a name="128611978"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128611978" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128611978">(Jun 25 2018 at 17:25)</a>:</h4>
<p>Ah, I assumed when you said initial that you were implying there were further populations. My bad.</p>



<a name="128612049"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128612049" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128612049">(Jun 25 2018 at 17:27)</a>:</h4>
<p>That code surface-level looks wrong to me (not panic safe)... but I don't obviously see how it could be related</p>



<a name="128612120"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128612120" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128612120">(Jun 25 2018 at 17:28)</a>:</h4>
<p>I don't think it is related, but on an initial glance I thought it might make the buffer too large if there was a data race or something like that due to the unsafe block.</p>



<a name="128612147"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128612147" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128612147">(Jun 25 2018 at 17:29)</a>:</h4>
<p>hm, I don't think so</p>



<a name="128612371"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128612371" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128612371">(Jun 25 2018 at 17:33)</a>:</h4>
<p>Now I'm installing Wireshark to see what the network data is</p>



<a name="128612675"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128612675" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128612675">(Jun 25 2018 at 17:38)</a>:</h4>
<p>So, that code is still being called. The <code>old_len</code> size does keep increasing (I kept setting increasingly larger and larger conditional breakpoints on line 213) and with a conditional breakpoint that will not be hit (checking for <code>old_len &lt; 0</code>), the memory usage does not increase (it is a lot slower, so I might just not have observed it yet).</p>



<a name="128612699"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128612699" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128612699">(Jun 25 2018 at 17:39)</a>:</h4>
<p>Isn't that expected behavior though? i.e., that we want to grow the buffer?</p>



<a name="128612707"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128612707" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128612707">(Jun 25 2018 at 17:39)</a>:</h4>
<p>At least I understood that code as intentionally growing the buffer</p>



<a name="128612753"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128612753" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128612753">(Jun 25 2018 at 17:40)</a>:</h4>
<p>It is, yeah.</p>



<a name="128612777"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128612777" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128612777">(Jun 25 2018 at 17:41)</a>:</h4>
<p>I think I may be jumping the gun, I think it's just increasing slowly.</p>



<a name="128612788"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128612788" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128612788">(Jun 25 2018 at 17:41)</a>:</h4>
<p>And I know that the buffer isn't shared between threads after having dug into that library a bit.</p>



<a name="128612792"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128612792" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128612792">(Jun 25 2018 at 17:41)</a>:</h4>
<p>(which makes sense, there's no reason for it to be shared)</p>



<a name="128613158"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128613158" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128613158">(Jun 25 2018 at 17:49)</a>:</h4>
<p><span class="user-mention" data-user-id="116107">@David Wood</span> are you using https locally too? can you use not?</p>



<a name="128613180"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128613180" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128613180">(Jun 25 2018 at 17:49)</a>:</h4>
<p>I'm only using http locally. It's not TLS by the looks of it.</p>



<a name="128613357"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128613357" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128613357">(Jun 25 2018 at 17:51)</a>:</h4>
<p>ah. I think my errors were coming from https (at least switching to http gets further)</p>



<a name="128613418"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128613418" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128613418">(Jun 25 2018 at 17:52)</a>:</h4>
<p>Yeah, normally mine will execute until the end (for my local Jenkins instances, it's around 27 mins).</p>



<a name="128613483"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128613483" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128613483">(Jun 25 2018 at 17:54)</a>:</h4>
<p><a href="/user_uploads/4715/KRLECS3vJCiqwZmCKEyk9Fto/pasted_image.png" target="_blank" title="pasted_image.png">pasted image</a> </p>
<div class="message_inline_image"><a href="/user_uploads/4715/KRLECS3vJCiqwZmCKEyk9Fto/pasted_image.png" target="_blank" title="pasted image"><img src="/user_uploads/4715/KRLECS3vJCiqwZmCKEyk9Fto/pasted_image.png"></a></div><p>Seems reasonably stable</p>



<a name="128613500"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128613500" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128613500">(Jun 25 2018 at 17:54)</a>:</h4>
<p>clearly you need to switch to macOS</p>



<a name="128613597"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128613597" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128613597">(Jun 25 2018 at 17:56)</a>:</h4>
<p>That's interesting - I wonder if is proportional to how many threads there are? I've had it happen on Linux and Windows (well, technically, inside WSL, but close enough).</p>



<a name="128613636"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128613636" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128613636">(Jun 25 2018 at 17:57)</a>:</h4>
<p>In fact, that might make sense - when I ran the original program to completion with heaptrack (it took 2 days and 18 hours..), it started to plateau at around 7GB.</p>



<a name="128613687"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128613687" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128613687">(Jun 25 2018 at 17:58)</a>:</h4>
<p>If a relationship between number of threads and maximum memory (and size of responses from Jenkins) holds, then I guess the question is, why does increasing the number of threads have a large (non-linear?) impact on memory usage of the threads?</p>



<a name="128613813"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128613813" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128613813">(Jun 25 2018 at 18:00)</a>:</h4>
<p>Running a single-instance, single-threaded process for each Jenkins instance would result in a great deal less memory being used (which is how I'm working around this for now, I'm really just curious why it is happening)</p>



<a name="128614357"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128614357" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128614357">(Jun 25 2018 at 18:12)</a>:</h4>
<p>I mean, I still think there's something truly wrong cause there was only like a few MB of data transferred. Where are 100MB coming form</p>



<a name="128614825"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128614825" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128614825">(Jun 25 2018 at 18:21)</a>:</h4>
<p>Right, it's not clear where the memory use is coming from</p>



<a name="128615031"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128615031" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128615031">(Jun 25 2018 at 18:25)</a>:</h4>
<p>When I run heaptrack, it shows the size of allocations as a bar chart. The vast majority of allocations are very small. For that reason I've always considered it a "why isn't it being dropped" rather than "why is it being allocated".</p>



<a name="128615048"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128615048" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128615048">(Jun 25 2018 at 18:25)</a>:</h4>
<p>Since I expected the allocations to be small, like the responses.</p>



<a name="128615099"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128615099" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128615099">(Jun 25 2018 at 18:26)</a>:</h4>
<p>Though, it occurs to me it could just be a few large allocations alongside a lot of small ones.</p>



<a name="128615116"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128615116" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128615116">(Jun 25 2018 at 18:26)</a>:</h4>
<p>BTreeMap is inherently lots of small allocations, IIRC</p>



<a name="128615138"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128615138" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128615138">(Jun 25 2018 at 18:27)</a>:</h4>
<p>And BTreeMap is where the majority of memory is spent according to massif</p>



<a name="128618799"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128618799" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128618799">(Jun 25 2018 at 19:41)</a>:</h4>
<p>I see each thread allocating ~400MB at "boot": </p>
<p><a href="/user_uploads/4715/Xy8QIN2b8uYJIkTJuPzPHFfh/pasted_image.png" target="_blank" title="pasted_image.png">pasted image</a> </p>
<div class="message_inline_image"><a href="/user_uploads/4715/Xy8QIN2b8uYJIkTJuPzPHFfh/pasted_image.png" target="_blank" title="pasted image"><img src="/user_uploads/4715/Xy8QIN2b8uYJIkTJuPzPHFfh/pasted_image.png"></a></div><p>It's all coming from <code>get_home</code>:</p>
<p><a href="/user_uploads/4715/TaZthCy8Ld6d6H1K_h_lB74C/pasted_image.png" target="_blank" title="pasted_image.png">pasted image</a> </p>
<div class="message_inline_image"><a href="/user_uploads/4715/TaZthCy8Ld6d6H1K_h_lB74C/pasted_image.png" target="_blank" title="pasted image"><img src="/user_uploads/4715/TaZthCy8Ld6d6H1K_h_lB74C/pasted_image.png"></a></div><p>There are 20997 entries in the jobs vector...</p>



<a name="128618812"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128618812" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128618812">(Jun 25 2018 at 19:41)</a>:</h4>
<p>Is that the same amount allocated at boot with no threads?</p>



<a name="128618970"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128618970" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128618970">(Jun 25 2018 at 19:45)</a>:</h4>
<p><a href="/user_uploads/4715/sfSpFD5sA-PSm6PYAX8Wfg5s/pasted_image.png" target="_blank" title="pasted_image.png">pasted image</a> </p>
<div class="message_inline_image"><a href="/user_uploads/4715/sfSpFD5sA-PSm6PYAX8Wfg5s/pasted_image.png" target="_blank" title="pasted image"><img src="/user_uploads/4715/sfSpFD5sA-PSm6PYAX8Wfg5s/pasted_image.png"></a></div><p>Looks like ~400 MB twice</p>



<a name="128618984"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128618984" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128618984">(Jun 25 2018 at 19:45)</a>:</h4>
<p>Huh, I wonder why it is twice.</p>



<a name="128619041"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128619041" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128619041">(Jun 25 2018 at 19:46)</a>:</h4>
<p>I guess that contributes to a bunch of the memory usage. Does it explain the memory increasing with threads?</p>



<a name="128619042"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128619042" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128619042">(Jun 25 2018 at 19:46)</a>:</h4>
<p>each instance is sequential w/o threads, yeah?</p>



<a name="128619046"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128619046" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128619046">(Jun 25 2018 at 19:46)</a>:</h4>
<p>Yeah.</p>



<a name="128619047"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128619047" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128619047">(Jun 25 2018 at 19:46)</a>:</h4>
<p>I guess it would be that.</p>



<a name="128619279"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128619279" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128619279">(Jun 25 2018 at 19:51)</a>:</h4>
<p>You could start scattering <a href="https://crates.io/crates/heapsize" target="_blank" title="https://crates.io/crates/heapsize">https://crates.io/crates/heapsize</a> annotations everywhere...</p>



<a name="128619809"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128619809" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128619809">(Jun 25 2018 at 20:02)</a>:</h4>
<p><span class="user-mention" data-user-id="116107">@David Wood</span> can you use nightly? I wonder if <a href="https://github.com/rust-lang/rust/pull/50352" target="_blank" title="https://github.com/rust-lang/rust/pull/50352">https://github.com/rust-lang/rust/pull/50352</a> would help you out a lot.</p>



<a name="128619852"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128619852" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128619852">(Jun 25 2018 at 20:03)</a>:</h4>
<p>If that was merged on 12th May, wouldn't it be in nightly and helping right now? Since we need to use nightly to use the system allocator in our profiles?</p>



<a name="128619902"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128619902" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128619902">(Jun 25 2018 at 20:04)</a>:</h4>
<p>Uhhhhh<br>
<em>shhh</em>.</p>



<a name="128619921"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128619921" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128619921">(Jun 25 2018 at 20:05)</a>:</h4>
<p>This explains why <code>cargo +nightly build</code> had nothing to do...</p>



<a name="128620001"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128620001" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128620001">(Jun 25 2018 at 20:06)</a>:</h4>
<p>I wonder if there's a way to improve the 56MB response from <code>get_home</code> turning into ~400MB of data structure.</p>



<a name="128620038"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128620038" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128620038">(Jun 25 2018 at 20:07)</a>:</h4>
<p><code>serde_json</code> taking up more space than the responses certainly explains the magnitude of the memory usage, just not the increasing behaviour?</p>



<a name="128620126"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128620126" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128620126">(Jun 25 2018 at 20:08)</a>:</h4>
<p>right, but I can't yet see the increasing behavior. It is interesting you saw it on multiple OS though. </p>
<p>You said WSL, right? I wonder what "system allocator" means there</p>



<a name="128620176"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128620176" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128620176">(Jun 25 2018 at 20:09)</a>:</h4>
<p>Yeah, all the data I've shown has been from WSL - that's where I primarily work. The linux machine it's ran on was the one where I discovered the issue when the program got killed.</p>



<a name="128620270"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128620270" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128620270">(Jun 25 2018 at 20:11)</a>:</h4>
<p>Would it be non-trivial to try Windows Real?</p>



<a name="128620280"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128620280" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128620280">(Jun 25 2018 at 20:12)</a>:</h4>
<p>but I suppose you did jemalloc and "system"...</p>



<a name="128620345"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128620345" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128620345">(Jun 25 2018 at 20:12)</a>:</h4>
<p><a href="https://github.com/mockersf/jenkins-api.rs/issues/19" target="_blank" title="https://github.com/mockersf/jenkins-api.rs/issues/19">https://github.com/mockersf/jenkins-api.rs/issues/19</a> could certainly help reduce the size of the requests - but as <code>?depth=1</code> is required to get the data I need, and that's incompatible with <code>tree</code> (oh, Jenkins...). Though, we don't need <code>?depth=1</code> for the <code>get_home</code> call - that makes it _way_ bigger and the library parses it as <code>ShortJob</code> anyway (throwing away a bunch of the data). So perhaps that is a improvement for reducing the usage. Again, doesn't explain the increasing behaviour.</p>



<a name="128620346"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128620346" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128620346">(Jun 25 2018 at 20:13)</a>:</h4>
<p>I can give it a go.</p>



<a name="128620878"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128620878" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128620878">(Jun 25 2018 at 20:23)</a>:</h4>
<blockquote>
<p>(throwing away a bunch of the data)</p>
</blockquote>
<p>But it doesn't, right? </p>
<div class="codehilite"><pre><span></span><span class="w">    </span><span class="cp">#[serde(flatten)]</span><span class="w"></span>
<span class="w">    </span><span class="k">pub</span><span class="p">(</span><span class="k">crate</span><span class="p">)</span><span class="w"> </span><span class="n">other_fields</span>: <span class="nb">Option</span><span class="o">&lt;</span><span class="n">serde_json</span>::<span class="n">Value</span><span class="o">&gt;</span><span class="p">,</span><span class="w"></span>
</pre></div>



<a name="128620941"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128620941" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128620941">(Jun 25 2018 at 20:24)</a>:</h4>
<p>Technically no, but there's no API (unlike <code>CommonJob</code>) to convert it to a type that will take advantage of all the fields.</p>



<a name="128621159"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128621159" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128621159">(Jun 25 2018 at 20:28)</a>:</h4>
<p>Then it seems like a terrible idea to have the <code>other_fields</code> field...</p>



<a name="128621194"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128621194" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128621194">(Jun 25 2018 at 20:29)</a>:</h4>
<p>There's no reason the author couldn't add a function that does it - but it would only have sufficient information to do it if the <code>Jenkins</code> instance was made with <code>?depth=1</code>.</p>



<a name="128621249"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128621249" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128621249">(Jun 25 2018 at 20:30)</a>:</h4>
<p>It seems like this still happens on Windows.</p>



<a name="128621270"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128621270" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128621270">(Jun 25 2018 at 20:30)</a>:</h4>
<p>huh. I can't fathom why I wouldn't see it on macos</p>



<a name="128621278"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128621278" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128621278">(Jun 25 2018 at 20:31)</a>:</h4>
<p>VMMap is actually pretty interesting - it shows each individual allocation and its size. I should be able to get traces too. But saving the data to a file makes it stop responding, so, there's that.</p>



<a name="128621309"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128621309" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128621309">(Jun 25 2018 at 20:31)</a>:</h4>
<p>I think it does happen on MacOS based on what you sent previously - it's just that when running with two instances, the maximum that it increases to is lower?</p>



<a name="128621356"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128621356" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128621356">(Jun 25 2018 at 20:32)</a>:</h4>
<p>I removed</p>
<div class="codehilite"><pre><span></span><span class="w">    </span><span class="cp">#[serde(flatten)]</span><span class="w"></span>
<span class="w">    </span><span class="k">pub</span><span class="p">(</span><span class="k">crate</span><span class="p">)</span><span class="w"> </span><span class="n">other_fields</span>: <span class="nb">Option</span><span class="o">&lt;</span><span class="n">serde_json</span>::<span class="n">Value</span><span class="o">&gt;</span><span class="p">,</span><span class="w"></span>
</pre></div>


<p>And it still compiles. So that be a thing to try.</p>



<a name="128621458"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128621458" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128621458">(Jun 25 2018 at 20:34)</a>:</h4>
<p>Yeah, I've got a few different ways to lower the overall memory usage now, I think.</p>



<a name="128621572"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128621572" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128621572">(Jun 25 2018 at 20:36)</a>:</h4>
<p>I do think there's a bug in Hyper / Tokio. It pretty reliably dies with </p>
<div class="codehilite"><pre><span></span>thread &#39;ubuntu1&#39; panicked at &#39;called `Result::unwrap()` on an `Err` value: Error { kind: Json(Error(&quot;Unexpected eof during chunk size line&quot;, line: 1, column: 58261504)), url: None }&#39;, libcore/result.rs:945:5
</pre></div>



<a name="128621598"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128621598" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128621598">(Jun 25 2018 at 20:37)</a>:</h4>
<p>Isn't that only because I call <code>unwrap()</code> everywhere in the repro?</p>



<a name="128621670"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128621670" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128621670">(Jun 25 2018 at 20:39)</a>:</h4>
<p>I mean, how is the data being corrupted?</p>



<a name="128621733"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128621733" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128621733">(Jun 25 2018 at 20:40)</a>:</h4>
<p>It wouldn't surprise me if the Jenkins instance was sending messed up data when asked for jobs that have _lots_ of data rather than it being a bug in Tokio/Hyper.</p>



<a name="128621739"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128621739" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128621739">(Jun 25 2018 at 20:40)</a>:</h4>
<p>heh, if thread2 gets the panic, thread1 can keep on going</p>



<a name="128621748"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128621748" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128621748">(Jun 25 2018 at 20:40)</a>:</h4>
<p>I spose I'd have to figure out how to use wireshark with HTTPS to find out for sure</p>



<a name="128621869"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128621869" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128621869">(Jun 25 2018 at 20:43)</a>:</h4>
<p>We're still not sure why it increases gradually though. There's an expected increase as all the <code>get_home</code> calls return, then it should be a straight line with spikes for the <code>CommonJob</code> allocations and frees - but there isn't, it just keeps going. I don't know how much memory gets allocated for the <code>get_home</code> call on my Jenkins instances, but over time it increases up to 7GB of memory, so it can't be from <code>get_home</code>, you can observe all the threads having passed that point.</p>



<a name="128622055"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128622055" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128622055">(Jun 25 2018 at 20:47)</a>:</h4>
<p>I'm trying with 2x threads. One thread died right away, but the other kept going. Over 50 jobs (~5 min), it allocated a net of +17.42 MB. Is that the rate you were seeing?</p>



<a name="128622075"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128622075" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128622075">(Jun 25 2018 at 20:47)</a>:</h4>
<p>I believe heaptrack said 20.4MB/s - but yeah, thereabouts.</p>



<a name="128622184"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128622184" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128622184">(Jun 25 2018 at 20:49)</a>:</h4>
<p>Unless you meant it allocated a total of 17.42MB. That's about what I'd see each second.</p>



<a name="128622323"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128622323" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128622323">(Jun 25 2018 at 20:52)</a>:</h4>
<p>a total, yes</p>



<a name="128622364"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128622364" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128622364">(Jun 25 2018 at 20:53)</a>:</h4>
<p>Which I'd actually expect that to be because my range doesn't include the next deallocation</p>



<a name="128622610"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128622610" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128622610">(Jun 25 2018 at 20:58)</a>:</h4>
<p><a href="/user_uploads/4715/coz4GyybN91eon5522kRpEgY/pasted_image.png" target="_blank" title="pasted_image.png">pasted image</a> </p>
<div class="message_inline_image"><a href="/user_uploads/4715/coz4GyybN91eon5522kRpEgY/pasted_image.png" target="_blank" title="pasted image"><img src="/user_uploads/4715/coz4GyybN91eon5522kRpEgY/pasted_image.png"></a></div><p>generations are roughly each iteration of the jobs <code>for</code> loop</p>



<a name="128622659"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128622659" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128622659">(Jun 25 2018 at 20:59)</a>:</h4>
<p>You're not seeing any allocations from the later loop iterations?</p>



<a name="128622892"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128622892" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128622892">(Jun 25 2018 at 21:03)</a>:</h4>
<p>Nope<br>
<a href="/user_uploads/4715/6hzH4wKl26HNaGo5B6202ahZ/pasted_image.png" target="_blank" title="pasted_image.png">pasted image</a></p>
<div class="message_inline_image"><a href="/user_uploads/4715/6hzH4wKl26HNaGo5B6202ahZ/pasted_image.png" target="_blank" title="pasted image"><img src="/user_uploads/4715/6hzH4wKl26HNaGo5B6202ahZ/pasted_image.png"></a></div>



<a name="128622905"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128622905" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128622905">(Jun 25 2018 at 21:03)</a>:</h4>
<p>Maybe I do need to switch to macOS.</p>



<a name="128622989"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128622989" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128622989">(Jun 25 2018 at 21:05)</a>:</h4>
<p>well, I can't get both threads to run at the same time, so maybe it's not all milk and honey</p>



<a name="128623449"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128623449" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128623449">(Jun 25 2018 at 21:15)</a>:</h4>
<p>Now I have a <code>Mutex</code> <em>and</em> a <code>Barrier</code>. Exciting</p>



<a name="128623590"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128623590" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128623590">(Jun 25 2018 at 21:18)</a>:</h4>
<p>haha. you'll love this — the memory usage is now going down over the iterations</p>



<a name="128623885"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128623885" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128623885">(Jun 25 2018 at 21:24)</a>:</h4>
<p><a href="/user_uploads/4715/9ppZg4nhBy6zlR8-k6lzB4iA/pasted_image.png" target="_blank" title="pasted_image.png">pasted image</a> </p>
<div class="message_inline_image"><a href="/user_uploads/4715/9ppZg4nhBy6zlR8-k6lzB4iA/pasted_image.png" target="_blank" title="pasted image"><img src="/user_uploads/4715/9ppZg4nhBy6zlR8-k6lzB4iA/pasted_image.png"></a></div><p>2 threads, 44 iterations. no memory increase</p>



<a name="128623922"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128623922" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128623922">(Jun 25 2018 at 21:25)</a>:</h4>
<p>It seems a lot like my local Jenkins data is causing a quite different effect.</p>



<a name="128623986"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128623986" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128623986">(Jun 25 2018 at 21:26)</a>:</h4>
<p><a href="/user_uploads/4715/4hg4ryvHojGtCpd2oVGk3cD-/pasted_image.png" target="_blank" title="pasted_image.png">pasted image</a>  This is my most recent profiling run with the ubuntu Jenkins.</p>
<div class="message_inline_image"><a href="/user_uploads/4715/4hg4ryvHojGtCpd2oVGk3cD-/pasted_image.png" target="_blank" title="pasted image"><img src="/user_uploads/4715/4hg4ryvHojGtCpd2oVGk3cD-/pasted_image.png"></a></div>



<a name="128623992"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128623992" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128623992">(Jun 25 2018 at 21:26)</a>:</h4>
<p>The first flag is 861MB , the second is 856MB</p>



<a name="128624015"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128624015" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128624015">(Jun 25 2018 at 21:26)</a>:</h4>
<p>That's with five copies of the instance.</p>



<a name="128624032"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128624032" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128624032">(Jun 25 2018 at 21:27)</a>:</h4>
<p>It definitely terminates quicker and doesn't go up as much as with my local Jenkins.</p>



<a name="128624038"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128624038" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128624038">(Jun 25 2018 at 21:27)</a>:</h4>
<p>In particular, the profiling data said <code>get_home</code> for that run.</p>



<a name="128624046"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128624046" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128624046">(Jun 25 2018 at 21:27)</a>:</h4>
<p>But my local Jenkins massif data is all <code>get_full_job</code>.</p>



<a name="128624099"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128624099" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128624099">(Jun 25 2018 at 21:28)</a>:</h4>
<p>huh. 5threads ~ 800 MB. Mine is 2threads ~800 MB.</p>



<a name="128624152"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128624152" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128624152">(Jun 25 2018 at 21:29)</a>:</h4>
<p>Well, I have no clue what's going on. Pretty sure there's a bug here, but I could tell you nothing about it.</p>



<a name="128626779"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128626779" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128626779">(Jun 25 2018 at 22:28)</a>:</h4>
<p>Currently running memory profile for my Jenkins instances in single-threaded and multi-threaded. For multi-threaded, I see the memory usage creep up over time, currently at 14.6% of my total memory. For the single-threaded, it doesn't grow, it got to a constant 0.5-0.7% and it stays around there. Just in writing this, multi-threaded has increased to 15.7% now. I'll post the massif data when it finishes. But this is enough to convince me that there is a problem and I'm not imagining things.</p>



<a name="128627723"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128627723" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128627723">(Jun 25 2018 at 22:56)</a>:</h4>
<p><a href="/user_uploads/4715/NmsD6T9t0anKfTc23WFMoLgG/massif-singlethreaded.out.18768" target="_blank" title="massif-singlethreaded.out.18768">massif-singlethreaded.out.18768</a> <a href="/user_uploads/4715/UNdpfWSxx4MdI_yQkvxLy6Kv/massif-multithreaded.out.18042" target="_blank" title="massif-multithreaded.out.18042">massif-multithreaded.out.18042</a></p>



<a name="128658225"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128658225" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128658225">(Jun 26 2018 at 14:09)</a>:</h4>
<p>So, I just got a chance today to look at the data from those profiles yesterday. Sometime after my comment here, the single-threaded memory usage spiked.</p>



<a name="128658258"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128658258" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128658258">(Jun 26 2018 at 14:10)</a>:</h4>
<p>I'm thinking that this isn't a threading issue at all. It's just that the threads highlight the problem quicker/more obviously.</p>



<a name="128658334"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128658334" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128658334">(Jun 26 2018 at 14:11)</a>:</h4>
<p>Or there might not be a problem at all, and this application just uses a bunch of memory in a strange-looking way. In which case, I'll have wasted your time, if that's the case - sorry about that.</p>



<a name="128664745"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128664745" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128664745">(Jun 26 2018 at 16:13)</a>:</h4>
<p>I'm now pretty confident having done a bunch more profiling that threads aren't an issue at all - I've got a Jenkins response that is giant (hundreds of megabytes) and that makes the memory usage spike something awful - I've seen it range from 1.9GB to 7GB total, and I'm not sure why there's a range, but I've confirmed that everything looks about normal without that Jenkins instance.</p>



<a name="128664807"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128664807" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128664807">(Jun 26 2018 at 16:14)</a>:</h4>
<p>FWIW, I did the digging I did because I wanted to learn a bit more about using Instruments for memory-related stuff, so no harm no foul</p>



<a name="128665708"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128665708" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128665708">(Jun 26 2018 at 16:31)</a>:</h4>
<p>(just as an FYI, it occurs to me the 7GB usage is explained by that job being huge too - in the actual application it goes further and uses fetches the builds of that job, they're also going to be giant, so the usage increases then too - to 7GB. In the smaller repro, it just uses 1.9GB because it is only the job.)</p>



<a name="128666268"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128666268" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128666268">(Jun 26 2018 at 16:42)</a>:</h4>
<p>I don't think any API should be sending back 10MB of JSON, let alone 100 or 1000MB</p>



<a name="128666467"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128666467" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128666467">(Jun 26 2018 at 16:46)</a>:</h4>
<p>I think it should be solvable with some PRs (though the author has started work on this functionality) to the <code>jenkins-api</code> crate so that it can use the <code>tree</code> query parameter instead of only <code>depth</code>.</p>



<a name="128666579"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128666579" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128666579">(Jun 26 2018 at 16:49)</a>:</h4>
<p>it's also possible that this could be a case where you may want to scatter some <code>drop</code> calls about, if you have "longer" methods</p>



<a name="128666684"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/davidtwco%27s%20memory%20issue/near/128666684" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/davidtwco&#x27;s.20memory.20issue.html#128666684">(Jun 26 2018 at 16:50)</a>:</h4>
<p>Perhaps, yeah. Regardless, thanks to both you (<span class="user-mention" data-user-id="116155">@Jake Goulding</span> ) and <span class="user-mention" data-user-id="116122">@simulacrum</span>  for spending some time looking into this.</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>